Expected Values

The Expected Value of a Random Variable $X$ is the sum of the product of each possible value¹ of $X$ times the Probability of seeing that value. In the discrete case (and you’ll see all these as ways of notating the same idea),

$$\begin{align*} E[X] &= \sum{xP(X=x)} \\ &= \sum{xP(x)} \\ &= \sum{xf_X(x)} \\ &= \sum{xf(x)} \end{align*}$$

Where $f_X(x)$ is the probability distribution² of $X$. You’d use the fancy $\int$ in place of the humble $\sum$ above in the continous case.

Linearity and Constancy

The Expected Value of a number 42 is… drumroll… 42. This may seem silly to state but there’s a Linearity property that’s pretty important. If $X$ is a Random Variable and $a$ and $b$ are some constants,

$$E[a + bX] = E[a] + E[bX] = a + bE[X]$$

This is a Very Nice Thing to use in proofs and computations. $a$ shifts and $b$ scales $E[X]$.

Important Things

$E[E[X]] = E[X]$. May seem obvious to a lot of people but not to yours truly because I overthink things. $E[X]$ has been computed and is not a Random Variable!

What is the Expected Value of a Dice Roll?
3.5
OK what is the Expected Value of the Expected Value of a Dice Roll?
We just did that. Still 3.5, the Expected Value of a Dice Roll… are you okay?

$\square$

Now this one’s a doozy: if $Y$ is another Random Variable, $E[E[Y|X]] = E[Y]$. How can that be?

Consider this: What is the Expected Value of height $H$ in 145 people you picked at random and where all heights are equally likely?

$$\begin{align*} E[H] &= \sum_{i=1}^{145}{h_i} \cdot P(H=h_i) \\ &= \sum_{i=1}^{145}{h_i} \cdot \frac{1}{145} \\ &= \frac{1}{145} \sum{h_i} \end{align*}$$

Now you ask: What is the Expected Value of the height given a Random Variable Sex, $S \in \{\text{Male}, \text{Female}, \text{Intersex}\}$? This is a simple conditional probability. Remembering the Expected Value is the sum of products of realizations and their probabilities,

$$E[H|S] = \sum{h \cdot P(H=h|S)}$$

Now $E[H|S]$ is still a random variable because we haven’t specified a value for $S$ (i.e., we haven’t ‘collapsed’ it to a specific thing like $E[H|S=\text{Female}]$). So once again, remembering that Expected Value is the sum of products of all values of $S$ and their probabilities,

$$\begin{align*} E[E[H|S]] &= \sum_{s\in{\{M,F,I\}}}\left[{\sum{h\cdot P(H=h|S=s)}}\right] \\ &= \sum{h \cdot P(H=h|S=\text{Male})} \\ &+ \sum{h \cdot P(H=h|S=\text{Female})} \\ &+ \sum{h \cdot P(H=h|S=\text{Intersex})} \end{align*}$$

So you’re getting the Expected Value of the height across everyone in $S$, which is simply $E[H]$ 🥳 This is very nice when we get to Variance and Covariance!

Variance and Covariance

The Variance of a Random Variable $X$ is how much we expect it to deviate from its Expected Value and is a Random Variable itself³. We square it first because we want a nice positive number.

$$\begin{align*} \text{Var(X)} &= E[(X - E[X])^2] \\ &= E[X^2 - 2 \cdot X \cdot E[X] + (E[X])^2] \\ &= E[X^2] - E[2 \cdot X \cdot E[X]] + E[(E[X])^2] \\ &= E[X^2] - 2 \cdot E[X] \cdot E[E[X]] + (E[X])^2 \\ &= E[X^2] - 2 \cdot E[X] \cdot E[X] + (E[X])^2 \\ &= E[X^2] - 2 \cdot (E[X])^2 + (E[X])^2 \\ &= E[X^2] - (E[X])^2 \\ \end{align*}$$

The Law of the Unconscious Statistician (LOTUS)

It’s simple (and highly useful) enough in practice. If $g(X)$ is some function of Random Variable $X$,

$$E[g(X)] = \sum[g(x).f_X(x)]$$

I do not completely understand the proof but here it is for reference.

Footnotes

1. ‘Realizations’ is the fancy word. Denoted by $x$, lowercase. ↩

2. I’ve seen $f(x)$ too as a shorthand. Note that $F_X(x)$ and $F(x)$ refer to the Cumulative Density Function. ↩

3. Any function of a Random Variable is a Random Variable. ↩